What is Mixnode?

Mixnode turns the web into a giant database!

In other words, Mixnode allows you to think of all the web pages, images, videos, PDF files, and other resources on the web as rows in a database table; a giant database table with trillions of rows that you can query using the standard Structured Query Language (SQL). So, rather than running web crawlers/scrapers you can write simple queries in a familiar language to retrieve all sorts of interesting information from this table of live data.

Mixnode turns the web into a giant database table with multiple columns.

Just like a regular database table, you are provided with several columns (a.k.a. fields) that represent different attributes of web resources such as URL, content, content type, content language, domain name, ... Additionally, Mixnode comes with hundreds of functions that you can use to further analyze the data in any way that you want. From parsing HTML/XML and JSON to handling date/time and processing text, there are numerous built-in functions to use directly in your queries.

As a simple example, using Mixnode, getting the URL and title of every web page from the web boils down to a simple SQL query:

select 
    url,
    string_between(content, '<title>', '</title>') as title
from
    resources
where
    content_type like 'text/html%'

Where the results will look similar to:

You can expand this query in any number of ways by utilizing the built-in columns and functions of Mixnode. For example, if you wanted to get the title of every English web page you could simply use a condition on the content_language column:

select 
    url,
    string_between(content, '<title>', '</title>') as title
from
    resources
where
    content_type like 'text/html%' and
    content_language = 'en'

Did you want the title and first paragraph of every English web page? The css_text_first function has you covered:

select 
    url,
    string_between(content, '<title>', '</title>') as title,
    css_text_first(content, 'p') as first_paragraph
from
    resources
where
    content_type like 'text/html%' and
    content_language = 'en'

Same query, but only on .net domains? You only need to use the url_etld column:

select 
    url,
    string_between(content, '<title>', '</title>') as title,
    css_text_first(content, 'p') as first_paragraph
from
    resources
where
    content_type like 'text/html%' and
    content_language = 'en' and
    url_etld = 'net'

Consider the question "Sort the English Wikipedia articles by length". All you need to answer this question is to use the order by clause:

select
    url,
    cardinality(words(content)) as article_length
from 
    resources
where 
    url_host = 'en.wikipedia.org' and
    url_abs_path like '/wiki/%'
order by article_length desc

By combining table columns and built-in functions you can practically analyze the web in an infinite number of ways. Additionally, you can integrate Mixnode with external data sources (e.g. sending and receiving data from Amazon S3) and create even more flexible queries.

Give it a try!

Mixnode allows you to focus only on what you need to get from the web and not how to get it. It is an end-to-end solution that takes you from question to answer with a simple query; you don't need to deploy web crawlers or run scrapers, you don't need to process raw data, and there are no "intermediate results".