If you run out of time and need some additional expert Pentaho resources of tools please contact us! We’d be happy to learn more about your project and let you know how we can help.Metadata Injection in Pentaho Data Integration There is always the risk with any optimization that the system performs worse than you anticipated, and you will need to troubleshoot what to do next. Test the transformations well before moving them to production! Build a test environment as close as possible to the production environment and make a good test plan that covers as many cases as possible. Of course, you can combine both strategies, scale up and scale out. For example, you must bear in mind that if you order the results on the slave servers, it will be necessary to group them on the master server so that the result is kept orderly. With this strategy you must always take into account the form of execution that you are using and the separation of data that you carry out.
The idea is that the master server is responsible for distributing the work to the other servers and consolidating the results that each one returns, thus distributing the workload and improving the total time. In this cluster, one server will officiate as a master server and the others will officiate as slave servers. To achieve this, it is necessary to configure a Cluster of Pentaho PDI servers. If your server resources are not enough to execute your process within the acceptable times for your business, it is possible to improve the processing capacity by increasing the number of servers. Scale out – run the job on a cluster of integration servers But watch out, do not create more copies than the number of CPUs you have available! 4. If you find that there are some steps of your work that are creating a “bottleneck” in your process, you can run multiple copies of these steps to lower the total time at the expense of increased resource consumption. Scale up – execute several copies of the steps of your work that consume more resources The first process will read from line 1 to the line that corresponds to 1 GB, the second process will do it from 1 GB to 2 GB and so on with the rest of the file. If you have a 4 GB file, you can fire 4 processes in parallel that will be distributed reading work. For example, in the case of text files, you can define what is read in parallel and how many reading processes are generated. Pentaho PDI provides mechanisms to parallelize access to data. The same applies if your data source is a Database in which case readings can be triggered in parallel taking care not to saturate the database server. This is also a valid approach if your data is on a disk battery (RAID) or specialized external devices such as SAN or NAS.
#Pdi pentaho data integration full
Take full advantage of the possibilities of reading data provided by your source systemįor example, if you are reading text files you could store these files on different hard drives and shoot readings in parallel to take better advantage of the available hardware. For example, you can keep in memory the records that you use as lookup, so your resources are not constantly reading this data. Another use that you can give to your memory is to increase the amount of records that are kept in the buffer between the different steps of your work and in this way improve the times. For example, do not allocate a Gigabyte of memory to process a text file of a few hundred lines. Adjust the memory usage parameters of your toolĪllocate enough memory to carry out the work, considering the memory used by other applications and the operating system of your server itself.
The challenge of processing these large volumes of data requires attention to every detail and the application of best practices whenever possible. With the passage of time, the volumes of data that we handle every day are growing exponentially and there is an ongoing need for integration tools like Pentaho PDI to process larger and larger volumes of day.Ĭurrently, we talk about Terabytes of information of the Gigabytes of some years ago or the Kilobytes of decades ago.
0 kommentar(er)
