Replication HowTo

From ADempiere
Jump to: navigation, search
This Wiki is read-only for reference purposes to avoid broken links.

Summary

In this page, I will walk through the steps to deploy replication on ADempiere.

Overview

I am trying to achieve a basic scenario where customers and orders captured at a branch or store are replicated to a central store and where changes made at the central store (via a web portal) may be replicated back to the branch or store.

Discovering Replication Terms

The Type of data Replication determines the direction of the data replication.

  • Reference means that the data in this system is read only
  • Local means that the data in this system is not replicated to other systems
  • Merge means that the data in this system is synchronised with the other system

Apache ActiveMQ

Apache ActiveMQ handles the replication. You can find out more about the project here: http://activemq.apache.org

Role of ActiveMQ

ActiveMQ resides on the server you export from (in order to ensure there is no communication link failure between the export system and this application. ADempiere exports changes to records as defined by you when these records are change by capturing the change via a special model validator and then sending xml as defined by you to the ActiveMQ server. The ActiveMQ server then keeps hold of the change records and provides them to any authorised client that listens to the specific topic. If the clients link fails, ActiveMQ keeps the change message until the client's connection is restored to the ActiveMQ server.

Configuring ActiveMQ

Please follow the instructions on the ActiveMQ website. I used ActiveMQ v 5.2.0.

ADempiere Configuration Overview

These are the steps to configure ADempiere for replication. The exporting system is the system we wish to export record changes from. The importing system is the system we want to import record changes to. You can configure a system to be both an importing and an exporting system.


Exporting System

  • Define an Export Processor Type (org.adempiere.process.rpl.exp.TopicExportProcessor)
  • Define an Export Processor
    • the host is the hostname or ip address of where you run the ActiveMQ server
    • the port is the port number on which you run ActiveMQ - usually 61616
    • Add the following parameters (search key/parameter value)
      • isDeliveryModePersistent / true - keeps the change message in the queue if a client becomes disconnected
      • timeToLive / 0 (or maybe 10000) - 0 = forever however it would be unusual if a message could not be delivered between ADempiere and a local ActiveMQ service within 10000ms - I have been advised to set this to 0.
      • protocol / tcp
      • clientID / example - the main point is that the client ID for each connection must be different (however, I have also been told it must be the same)
      • topicName / exampleTopic - this need to be the same on the ImportProcessor
  • Define a replication strategy incorporating the export processor define above
    • Under Replication Table, select the tables you wish to replicate including the replication type
  • Select the replication strategy in Client window

Testing Export

Now you can test to see if your replication export is running

  • Open up the ActiveMQ monitor on http://yourhost:8161/admin
  • Check the topics do not include your exampleTopic
  • fire up ADempiere and make a change to one of the tables you defined as a replication table under replication strategy
  • check the topics again and see if your topic has been added to the list of topics in ActiveMQ and that there is 1 message sent n the topic. cYou will not have any consumers registered at this point - but we are ready to move on to Import Processors

If you have a problem, then check the following:

  • attach a debugger and set a break point in xxxx to see if the export is triggered

Importing System

NOTE: Changes to the import processor do not take effect until you restart the ADempiere server.

  • Configure an Import Processor Type using the class org.adempiere.server.rpl.imp.TopicImportProcessor
  • Configure an Import Processor with the following parameters
    • host = the remote hostname or ip address (of the Apache ActiveMQ server)
    • port = the port on the activeMQ server (usually 61616)
    • account = system, password info = manager
      • topicName =
      • protocol = tcp
      • isDurableSubscription = ture (this means you can disconnect and then reconnect - and the messages will be stored for you)
      • subscriptionName = Do not know - still looking at what this does
      • clientID = hostname


Current Issues

  • Replication only works point to point - I believe Victor is working on some changes to make the Replication work multicast
    • Routing of JMS message is administrated on JMS Server. Also JMS Topics allow many subscribers. trifon 07:16, 23 September 2009 (PDT)
  • You currently need to specify the ExportProcessor explicitly as an additional client model validator however imho this should be automatically loaded when you associate a replication strategy with a client
    • This can be done, but it is just request for improvement. No Issue i think. trifon 07:16, 23 September 2009 (PDT)
  • The version number is currently hard coded as 3.2.0
    • Version number is needed if ADempiere Export Formats with different versions exist in the system. trifon 07:16, 23 September 2009 (PDT)
  • It would be good if the exporting server was able to send the export format either with the record or to the other server the first time it starts to communicate with a new server. This would alleviate the issues with having mis-matched export formats on different servers causing the import of records to fail.
    • Sounds like interesting proposal. But this can be done and now. Administrator can define that Export Forat is exported and then all export formats defined on the Central Server will be replicated to the remote Servers. trifon 07:16, 23 September 2009 (PDT)
  • The search key of Export format needs to match (including case sensitivity) of the Column. This probably should be hard linked or refactored so that the search key always equals the column name. Actually not sure about this - I think this real issue is they must match so that the field in the Export Format match the Export Format used by the Importing system.
    • I do not remember to develop such restriction? Can you show an example? trifon 07:16, 23 September 2009 (PDT)
  • When you make a change to the export format - you need to reboot the application server so that it is loaded - there needs to be another way where the export format can be reloaded dynamically.
    • No need to restart application Server. You can stop/start ADempiere Export processor from ADempiere web Admin. trifon 07:16, 23 September 2009 (PDT)
  • You may receive an error saying these certain fields are not updateable - so if you do - you need to flag these fields as updateable in the AD.
    • Can you show example? This is possible, but this is limitation to the model in ADempiere. Replication uses Model classes from ADempiere to set values. trifon 07:16, 23 September 2009 (PDT)

Export Formats (Lines)

exported with psql:

psql -d adempiere -U adempiere -c 'select e.exp_formatline_id, e.value, e.name, ef.name as format_name, e.position, e.ismandatory, e.type, c.name as column_name, e.exp_embeddedformat_id, \
   e.ispartuniqueindex, e.dateformat from exp_formatline e join exp_format ef on e.exp_format_id = ef.exp_format_id join ad_column c on c.ad_column_id = e.ad_column_id \
   order by ef.name, e.position;' > mj.sql
exp_formatline_id |         value          |         name          |           format_name           | position | ismandatory | type |      column_name       | exp_embeddedformat_id | ispartuniqueindex |     dateformat      
------------------+------------------------+-----------------------+---------------------------------+----------+-------------+------+------------------------+-----------------------+-------------------+---------------------
            50041 | value                  | Value                 | All BPartners for Client/Tenant |       10 | N           | E    | Search Key             |                       | Y                 | 
          1000011 | AD_Client_ID           | Client                | BPartner Location               |       10 | Y           | R    | Client                 |                 50000 | N                 | 
          1000012 | C_BPartner_Location_ID | ID                    | BPartner Location               |       20 | Y           | E    | Partner Location       |                       | Y                 | 
          1000013 | C_BPartner_ID          | Business Partner      | BPartner Location               |       30 | Y           | R    | Business Partner       |                 50007 | N                 | 
          1000014 | C_Location_ID          | Location              | BPartner Location               |       40 | Y           | E    | Address                |                       | N                 | 
          1000015 | Phone                  | Phone                 | BPartner Location               |       50 | N           | E    | Phone                  |                       | N                 | 
          1000016 | Fax                    | Fax                   | BPartner Location               |       60 | N           | E    | Fax                    |                       | N                 | 
            50037 | client                 | Client ID             | Business Partner                |       10 | Y           | R    | Client                 |                       | Y                 | 
            50027 | AD_Client_ID           | Client                | Business Partner                |       10 | Y           | R    | Client                 |                 50000 | Y                 | 
            50038 | Value                  | Search key            | Business Partner                |       20 | Y           | E    | Search Key             |                       | Y                 | 
            50028 | AD_Org_ID              | Organization          | Business Partner                |       20 | Y           | R    | Organization           |                 50001 | N                 | 
            50039 | name                   | Name                  | Business Partner                |       30 | N           | E    | Name                   |                       | N                 | 
            50029 | Value                  | Business Partner Key  | Business Partner                |       30 | Y           | E    | Search Key             |                       | Y                 | 
            50030 | Name                   | Business Partner Name | Business Partner                |       40 | Y           | E    | Name                   |                       | N                 | 
            50040 | language               | Language              | Business Partner                |       40 | N           | E    | Language               |                       | N                 | 
            50031 | DUNS                   | DUNS                  | Business Partner                |       50 | N           | E    | D-U-N-S                |                       | N                 | 
            50032 | C_BP_Group_ID          | BP Group              | Business Partner                |       52 | Y           | R    | Business Partner Group |                 50003 | N                 | 
            50005 | Value                  | Value                 | Business Partner Group          |       10 | Y           | E    | Search Key             |                       | Y                 | 
            50006 | AD_Client_ID           | Client key            | Business Partner Group          |       30 | Y           | R    | Client                 |                 50000 | Y                 | 
            50000 | AD_Client_Value        | AD_Client_Value       | Client Value                    |       10 | N           | E    | Search Key             |                       | Y                 | 
          1000009 | AD_Client_ID           | Client                | Location                        |       10 | Y           | R    | Client                 |                 50000 | N                 | 
          1000050 | AD_Org_ID              | Organisation          | Location                        |       20 | Y           | R    | Organization           |                 50001 | N                 | 
          1000010 | C_Location_ID          | Location              | Location                        |       30 | Y           | E    | Address                |                       | N                 | 
          1000021 | Address1               | Address1              | Location                        |       40 | N           | E    | Address 1              |                       | N                 | 
          1000022 | Address2               | Address2              | Location                        |       50 | N           | E    | Address 2              |                       | N                 | 
          1000023 | City                   | City                  | Location                        |       60 | N           | E    | City                   |                       | N                 | 
          1000024 | Postal                 | Postal                | Location                        |       70 | N           | E    | ZIP                    |                       | N                 | 
          1000025 | Postal_Add             | Postal_Add            | Location                        |       80 | N           | E    | -                      |                       | N                 | 
          1000026 | C_Country_ID           | Country               | Location                        |       90 | Y           | E    | Country                |                       | N                 | 
          1000027 | C_Region_ID            | Region                | Location                        |      100 | N           | E    | Region                 |                       | N                 | 
          1000028 | C_City_ID              | City                  | Location                        |      110 | N           | E    | City                   |                       | N                 | 
          1000029 | RegionName             | Region Name           | Location                        |      120 | N           | E    | Region                 |                       | N                 | 
          1000030 | Address3               | Address3              | Location                        |      130 | N           | E    | Address 3              |                       | N                 | 
          1000031 | Address4               | Address4              | Location                        |      140 | N           | E    | Address 4              |                       | N                 | 
          1000059 | IsActive               | IsActive              | Location                        |      190 | N           | E    | Active                 |                       | N                 | 
            50010 | AD_Client_ID           | Client                | Order                           |       10 | Y           | R    | Client                 |                 50000 | N                 | 
            50011 | AD_Org_ID              | Organization          | Order                           |       20 | Y           | R    | Organization           |                 50001 | N                 | 
            50012 | DocumentNo             | Document number       | Order                           |       30 | Y           | E    | Document No            |                       | Y                 | 
            50017 | GrandTotal             | Grand Total           | Order                           |       80 | N           | E    | Grand Total            |                       | N                 | 
            50019 | C_DocType_ID           | Document Type         | Order                           |      100 | Y           | E    | Document Type          |                       | Y                 | 
            50020 | C_BPartner_ID          | Business Partner      | Order                           |      110 | Y           | E    | Business Partner       |                       | Y                 | 
            50007 | DocumentNo             | Document number       | Order key                       |       10 | N           | E    | Document No            |                       | Y                 | 
            50008 | C_DocType_ID           | Document Type         | Order key                       |       20 | Y           | E    | Document Type          |                       | Y                 | 
            50009 | C_BPartner_ID          | Business Partner      | Order key                       |       30 | Y           | E    | Business Partner       |                       | Y                 | 
            50021 | AD_Client_ID           | Client                | Order Line                      |       10 | Y           | R    | Client                 |                 50000 | N                 | 
            50022 | AD_Org_ID              | Organization          | Order Line                      |       20 | Y           | R    | Organization           |                 50001 | N                 | 
            50023 | M_Product_ID           | Product               | Order Line                      |       30 | N           | E    | Product                |                       | N                 | 
            50024 | PriceEntered           | Price Entered         | Order Line                      |       40 | N           | E    | Price                  |                       | N                 | 
            50025 | C_Order_ID             | Order                 | Order Line                      |       50 | N           | R    | Order                  |                 50004 | Y                 | 
            50026 | Line                   | Line number           | Order Line                      |       60 | Y           | E    | Line No                |                       | Y                 | 
            50001 | Value                  | Value                 | Organization                    |       10 | N           | E    | Search Key             |                       | Y                 | 
            50002 | AD_Client_ID           | Client                | Organization                    |       20 | N           | R    | Client                 |                 50000 | Y                 | 
          1000051 | AD_Client_ID           | Client                | Registration                    |       10 | Y           | R    | Client                 |                 50000 | Y                 | 
          1000052 | AD_Org_ID              | Organisation          | Registration                    |       20 | Y           | E    | Organization           |                       | N                 | 
          1000053 | A_Registration_ID      | Registration          | Registration                    |       30 | Y           | E    | Registration           |                       | Y                 | 
          1000054 | Name                   | Name                  | Registration                    |       40 | N           | E    | Name                   |                       | N                 | 
          1000055 | M_Product_ID           | Product               | Registration                    |       50 | N           | E    | Product                |                       | N                 | 
          1000056 | IsInProduction         | In Production         | Registration                    |       60 | N           | E    | In Production          |                       | N                 | 
          1000057 | IsRegistrered          | Registered            | Registration                    |       70 | N           | E    | Registered             |                       | N                 | 
          1000058 | AssetServiceDate       | Service Date          | Registration                    |       80 | N           | E    | In Service Date        |                       | N                 | 
          1000038 | AD_Client_ID           | Client                | User                            |       10 | Y           | R    | Client                 |                 50000 | Y                 | 
          1000060 | AD_Org_ID              | Organisation          | User                            |       15 | Y           | R    | Organization           |                 50001 | N                 | 
          1000039 | C_BPartner_ID          | Business Partner      | User                            |       20 | Y           | E    | Business Partner       |                       | Y                 | 
            50003 | Name                   | Name                  | User                            |       20 | Y           | E    | Name                   |                       | Y                 | 
          1000040 | Name                   | Name                  | User                            |       30 | Y           | E    | Name                   |                       | Y                 | 
            50004 | AD_Client_ID           | Client key            | User                            |       30 | Y           | R    | Client                 |                 50000 | Y                 | 
          1000041 | EMail                  | Email                 | User                            |       40 | N           | E    | EMail Address          |                       | N                 | 
          1000042 | Password               | Password              | User                            |       50 | N           | E    | Password               |                       | N                 | 
          1000043 | Birthday               | Birthday              | User                            |       60 | N           | E    | Birthday               |                       | N                 | yyyy-MM-dd hh:mm:ss
          1000044 | Phone                  | Phone                 | User                            |       70 | N           | E    | Phone                  |                       | N                 | 
          1000045 | Fax                    | Fax                   | User                            |       80 | N           | E    | Fax                    |                       | N                 | 
          1000061 | IsActive               | IsActive              | User                            |      130 | N           | E    | Active                 |                       | N                 |