This is about why we should not use flat files anymore:
Integration across systems in the 1990´s looked like this:
– Sender System runs export program in batch to produce flat file
– Transfer script sends flat file via FTP to Receiver System
– Receiver System runs import program in batch to update database
What is so bad about it:
– STORAGE: to store files on a (local) file system is not good from a Security and Auditing point of view. Any intruder who is able to access the file system, can read the data (unless they are right away encrypted, e.g. via PGP). It is better to persist your (especially sensitive) data in a database.
– TRANSFER: the transmission protocol FTP is not very secure and sometimes can not be so reliable (depending on the FTP server implementation). To secure the transmission use at least SSH/SSL encryption via SFTP/FTPs
– FORMAT: there are issues with flat files in general. The change management is horrible if you want to extend something (all senders and receivers have to switch their export/import programs in one shot) – and believe me: there will always be changes…
The only advantage is: File Size.
Nothing is leaner than a flat file, especially if you deal with high volumes/large messages.
But especially in SAP projects we learned already: Sometimes it is better to go a structured and organized way instead of choosing the light-weight and easy approach.
Options to improve existing integration flows (keeping the flat file format for the time being or for message size reasons):
– Clean up the SAP side first where you can instead of redesigning the flow completely: use ABAP Proxy with flat file as attachment to avoid the STORAGE problem and solve the TRANSFER issue by using http(s):
– TRANSFER: No more FTP to/from SAP servers, only RFC or HTTP
– STORAGE: Change your programs to write files into memory only and then pass the data to an ABAP proxy. This is of course only possible for you own programs, but SAP is also modernizing the way to transfer standard-based files.
For new integrations simply try to avoid flat files. They are not state-of-the-art anymore.
