Working as a Forward Deployed Engineer at Palantir was an incredibly rewarding opportunity. I was a part of a small team that was tasked with monitoring our Fortune 50 client’s network, to identify any malicious activity. We built a pipeline that processed terabytes of data from firewalls, intrusion detections systems, Active Directory, System Center Configuration Manager, and other data sources.
Much of my time was spent working with industry standard big data tools, like Apache Spark (pyspark), Hadoop, and ElasticSearch. I generated both recurring metrics, and ad hoc insights. These insights included sources of leaked credentials, vulnerable assets, and anomalous software.