This is a submission for the AI Agents Challenge powered by n8n and Bright Data
What I Built
I built an AI agent that aggregates and analyzes sentiment from:
- All online sources
- Major US stock market news sites
- X.com user posts
It delivers a consolidated Wall Street sentiment report in seconds. The agent scans trending trader discussions, financial headlines, and online chatter, then generates a concise, actionable summary—sent directly via email—so investors can understand market sentiment instantly without spending hours researching.
Demo
n8n Workflow
GitHub – https://github.com/wimalasuriyaib/WallStreetSentimentAnalyzer
Wall Street Sentiment Analyzer – This is my main Agent workflow.
Online Stock Market Sentiment Workflow
X.com Stock Market Sentiment Workflow
Agent Capabilities
Final ouput – US Stock Market Sentiment Report
Technical Implementation
Overview
The workflow is designed to automatically fetch real-time US stock market sentiment, process the data, and generate a concise word summary suitable for a blog post. It leverages BrightData for data extraction, Google AI for querying, and Google Gemini (PaLM API) for text summarization. The workflow is fully automated and orchestrated using n8n, an open-source workflow automation tool.
System Instructions
Triggering: Manual execution via the Manual Trigger node or can be scheduled with a Cron node.
Data Collection: BrightData Web Scraper Node: Sends a query to the BrightData dataset API to extract market sentiment from a specified URL using a predefined prompt
Snapshot Monitoring: The workflow waits for the BrightData snapshot to be ready and monitors its progress using the Check Snapshot Status node.
Data Handling: Once the snapshot is ready, the Download Snapshot Content node retrieves the data.
Edit Fields Node: Normalizes the output JSON and extracts the relevant answer_text for further processing.
Data Summarization: Google Gemini Node: Passes the extracted text to the Gemini model (models/gemini-2.0-flash) to generate a concise word summary suitable for a blog post.
Prompts are dynamically injected from the snapshot content for contextual summarization.
Model Choice
Google Gemini (PaLM API) – Selected for its ability to generate human-like, high-quality text summaries and handle complex financial language and sentiment analysis effectively.
Memory / Data Handling
Workflow uses pinData to store intermediate data (answer_text) securely within n8n.
Each node is stateless, relying on BrightData snapshots to maintain consistency and reproducibility.
Workflow design ensures error handling via conditional checks (IF nodes) to retry waiting or snapshot download until data is ready.
Tools Used
n8n: Orchestrates the workflow, manages triggers, and passes data between nodes.
BrightData: Handles data extraction from dynamic websites using snapshots. Provides monitoring APIs to ensure completeness and accuracy.
Google Gemini (PaLM API): Processes raw sentiment text. Produces coherent, concise summaries ready for blog publishing.
Workflow Highlights
- Asynchronous snapshot handling: Ensures the workflow doesn’t fail if data isn’t ready immediately.
- Dynamic prompt injection: Allows custom queries without modifying the workflow logic.
- Seamless integration: BrightData and Google Gemini nodes are fully credentialed and reusable for multiple datasets or sentiment sources.
- Scalable design: Can be extended to multiple stock tickers, social media sentiment, or regional markets by adjusting the query parameters.
Future Enhancements
- Integrate automated blog publishing via WordPress or Medium APIs.
- Add historical sentiment tracking and trend analysis.
- Incorporate alerts or notifications if sentiment changes drastically.
Bright Data Verified Node
The Bright Data Verified Node is a critical component in our stock market sentiment workflow, providing reliable and scalable web data extraction without the typical challenges of web scraping. By leveraging Bright Data, we can trigger dataset snapshots, monitor their progress in real-time, and download structured results automatically. This eliminates the need for building custom scraping pipelines, handling IP rotation, or managing proxy networks—tasks that are notoriously error-prone and time-consuming.
Using Bright Data ensures high data accuracy and compliance, which is particularly important when accessing dynamic and frequently updated sources like Google AI search results. Without it, we would face the complexity of dealing with anti-bot mechanisms, frequent source changes, and the overhead of continuously maintaining scraping scripts. Such manual approaches often result in inconsistent data, higher failure rates, and significant delays in processing, all of which could compromise the quality of downstream AI analysis.
By integrating the Verified Node, our solution gains reliability, speed, and maintainability. The node abstracts away the operational burdens of web data extraction, allowing us to focus on extracting insights, summarizing market sentiment with AI, and generating actionable content. Bright Data, therefore, transforms what could be a fragile, labor-intensive process into a seamless, scalable workflow.
Journey
Participating in this hackathon was an incredibly exciting experience, as it was my first time using both n8n and Bright Data. I began by spending several hours watching n8n videos. Since n8n is a low-code solution, my initial approach was to jump straight into building—but I quickly realized I lacked the basics and failed miserably after a few hours. While it is designed to accelerate development, mastering the fundamentals is essential.
I then went through the n8n Beginner Course on YouTube at high speed and complemented it by using freely available templates to build small projects for hands-on practice. I took a similar approach with Bright Data, experimenting with small projects to get comfortable with its capabilities.
Once I felt confident with both tools, I defined my problem statement: capture Wall Street sentiment analysis in seconds. Developing this stock market sentiment workflow was both challenging and rewarding. The initial goal was to capture real-time investor sentiment reliably and convert it into actionable AI-driven insights. A major hurdle was handling dynamic web content, especially from sources like Google AI Search, which frequently change and block automated requests. Without a robust solution, scraping would have been slow, error-prone, and difficult to maintain.
Integrating Bright Data’s Verified Node was a game-changer. It provided a secure, compliant, and scalable way to trigger dataset snapshots, monitor progress, and retrieve structured results effortlessly. This eliminated the need to manually manage proxies, IP rotations, and anti-bot measures.
Processing large amounts of unstructured text data was another challenge. Leveraging Google Gemini (PaLM) for summarization enabled us to convert raw responses into concise, high-quality 200-word blog posts. Combining Bright Data’s reliability with AI-powered summarization streamlined the workflow and significantly reduced operational complexity.
Since Bright Data can’t be directly added as an agent tool, I created two separate workflows: one to gather sentiment from online content and another from users on x.com. It took me some time to figure this out, but once implemented, completing the project became much faster.
The Hackception: Mini Hackathon Inside the Hackathon
A major highlight of this hackathon was involving students from the University of Peradeniya. During the Code to Cloud training program, I decided to run a mini hackathon within the main hackathon, introducing students to n8n and Bright Data. We launched it on Friday (just two days to go), conducted walkthroughs of sample projects, and then let students develop their own workflows. So far, one student has submitted their project, and I expect more submissions as the deadline approaches. To make it more exciting, we offered two free tickets to AWS Community Day Sri Lanka for students who delivered strong projects.
Check out my demo video on YouTube:
This journey reinforced the value of automation, scalability, and robust integrations, allowing me to focus on insights rather than infrastructure. Working with n8n was empowering, enabling rapid development of agentic solutions, while Bright Data simplified web data collection immensely. Overall, I gained deep technical knowledge, built a functional stock sentiment workflow, and successfully ran a hackathon inside a hackathon, inspiring the next generation of tech enthusiasts. I couldn’t ask for a more fulfilling experience.