Atypical Data Access using n8n

Client's data just out of reach? Here's how you get to it with n8n!

Way back in my very early days of IT, I was tasked to get network access to a different building. As I researched the options, it quickly became apparent that anything I would get someone else to do would be cost-prohibitive.

This is where I had to figure out a way to leverage what I had at my disposal to get this project to completion on time and under budget. The final solution? Run the network signal over the barbed wire fence that ran between the two buildings (you need to remember this was in the early 1990s)! We did experience a bit of packet loss every time a cow touched the fence but nothing major!

Sometimes you need to get creative if you don't have access to what you specifically need at the moment. This is a situation that many n8n users find themselves in when the systems they are trying to connect don't already have a node written for them, and they need to figure out unique ways of getting data to and from n8n.

This article is for those who find themselves in this position and aren't sure where to start. Hopefully, this will give you some ideas and inspiration.

HTTP Requests

OK. I'll admit, this may not be that atypical for those of you familiar with APIs, but if you have primarily stuck to nodes designed for specific purposes, this may be new to you. Just because there isn't a node for the specific service you want to use, it doesn't mean that you can't interact with data using n8n.

The HTTP Request node allows you to query any web service but is most commonly used to interact with APIs. If there isn't a node built into n8n, the next thing that I look for is an API for that service. I grab an HTTP Request node and see what I can get from the API if one exists.

Webhooks

Sometimes systems don't like you pulling information from them, but they don't mind pushing information to you. This is where a webhook comes in handy. A webhook essentially lets you receive data in n8n that is sent by another system. Once you have received the data from the webhook, you can then process it as you see fit.

One of the big advantages of using a webhook is that you can receive data in real-time from a system. So rather than constantly checking for updates or new information, you can receive the information and run your workflow only when new information is available.

Email

I recently needed to recover information from a system with such a bad API, and it was easier not to use it. I got the system admin to create a scheduled job that would extract the information I needed to an Excel spreadsheet and then email it to a specific email address.

I then configured n8n to check that email address once a day, retrieve the attached spreadsheet, and process the attached data. Once it was all set up, it worked like a charm!

RSS Feeds

While RSS feeds are becoming less and less popular, there was a time that they were the way to receive alerts and keep up to date with information. Just because they aren't popular anymore doesn't mean that they aren't useful!

Check the remote system to see if it has the ability to broadcast the information as an RSS feed. Then, point n8n at the RSS feed and process the information as it becomes available.

Shared Cloud Storage

Sometimes, all it takes is putting the data into cloud storage and sharing the file with someone else's account.

For example, if the system has the ability to save a CSV file to Dropbox and then share that file via Dropbox to your account, you can then have n8n monitor that file in Dropbox and process the data whenever there is a change in the file.

Sometimes, you don't even need to use cloud storage. If you have a file in Google Sheets, you can share the sheet with some else, and they can then access the data in the sheet from their Google Sheets.

FTP

Much like RSS feeds, we don't hear a lot about FTP anymore. Once the main way that people transferred files over the internet, cloud storage has largely taken its place.

But, it is still a legitimate way to move data and information between systems and n8n fully supports using FTP. If the system has an FTP server or can move files to an FTP server, n8n can do the rest.

SSH

If you are a Linux person, you are probably familiar with SSH. Typically used as a remote command-line system for managing servers, many are unaware that SSH can also transfer files.

By putting files on a system that has a properly configured SSH server, n8n can reach out to that server and move the data files from there.

MQTT & AMQP

MQTT and AMQP are messaging protocols that let you send information to a messaging queue where it waits to be picked up by another system. If your system that holds the data has the ability to send information using MQTT or AMQP, you can use n8n to retrieve and process it.

The only catch is that you will need to have a server available to broker the messages. Fortunately, that is not too difficult as both MQTT and AMQP are open-source protocols that are readily available!

Messaging Apps

n8n has support for several messaging apps built right into the system. Discord, Mattermost, Slack, and Telegram are just a few of the messaging apps available to n8n out of the box.

n8n can use these apps to send and receive information, much like MQTT and AMQP.

Database Access

Sometimes, you can access the data directly from the database using n8n. Microsoft SQL, MySQL and Postgres are all supported directly by n8n. If you are up on your SQL language, this might be the way to go.

Local File Mount

If you are running n8n locally inside your network, you might be able to access your data by mounting a network file share right on your n8n server. Regardless if n8n is installed on Windows or Linux, it is generally trivial on either of these operating systems to mount remote network shares.

Once the network shares are mounted, you can then access the data as simply files on the local n8n server.

3rd Party Tools on Self-Hosted n8n Server

Much like mounting remote file shares on your n8n server, you can also run other applications on your self-hosted n8n server that can act as an intermediary between n8n and your data.

Applications like syncthing, Hypercore Protocol, and rsync allow you to transfer or synchronize data, allowing you to process it with n8n.

2nd n8n Server Behind Firewall

Sometimes the problem is that the data you need to process is behind a firewall, and you can't get to it without making significant changes to the firewall. One solution is to put an on-premise copy of n8n behind the firewall to process data locally. This way, data stays safe behind the firewall, reducing the risk of a data leak.

Implementing a local n8n server can be accomplished in several ways:

  1. Install n8n on a small computer such as a Raspberry Pi or an Intel NUC
  2. Repurpose an old workstation at the company and install a Debian based OS and n8n on that system
  3. Install n8n inside a virtual computer on the company's virtualization platform

Conclusion

As you can see, there are many different options available to you for getting data to n8n for processing.

Great! You've successfully subscribed.
Great! Next, complete checkout for full access.
Welcome back! You've successfully signed in.
Success! Your account is fully activated, you now have access to all content.