NewsSocial WAIR

Web AI Resource

URL to Scrape:

Enter the full URL of the webpage you want to scrape.

Title Selector (Optional):

Enter CSS selector for the page title. If not provided, we'll try to find the title automatically.
Examples: .article-title, #post-title, h1.main-title, article h1

Custom CSS Selector (Optional):

Enter CSS selectors to extract specific elements. Multiple selectors can be separated by commas.
Examples: .article-content, #main-content, article p, .post-title

Exclude CSS Selector (Optional):

🚫 Auto-exclude common elements (footer, navbar, ads, etc.)

Enter CSS selectors for elements you want to exclude from the results. Multiple selectors can be separated by commas.
Examples: .ads-wrapper, .navigation-menu, .sidebar, #cookie-notice

Use Browser Engine

Enable this to use a real browser engine for JavaScript-rendered content.

Verify SSL Certificates

Uncheck this only if you're having SSL certificate issues with the target site.

🕷️ Enable Smart Crawling

Advanced Feature: When enabled, the scraper will follow links found on the main page and scrape each linked page (one level deep only).
⚠️ Warning: This may take significantly longer and make many requests.
Perfect for news sites to get full article content for each news item.
ℹ️ Note: Browser Engine will be automatically enabled for Smart Crawling.

📰 Extract Article Content

New Feature: When enabled, the scraper will intelligently extract the main article content and primary image from individual news articles.
ℹ️ Perfect for individual news articles (e.g., news/12345) to get clean content and main image.
Uses advanced content detection algorithms including semantic HTML analysis and readability scoring.
✨ Automatically detects article URLs and enables this feature when appropriate.

📝 Extract WordPress JSON Data

WordPress API Feature: When enabled, the scraper will extract data from WordPress JSON API endpoints.
ℹ️ Perfect for WordPress sites - automatically detects and extracts from wp-json/wp/v2/posts endpoints.
You can enter either the site URL (e.g., https://example.com) or direct wp-json URL (e.g., https://example.com/wp-json/wp/v2/posts?per_page=5&_embed).
✨ Automatically enables Browser Engine for better JSON extraction.

Example Selectors

.article-content - Elements with class "article-content"
#main-content - Element with ID "main-content"
h1 - All h1 headings
.post .title - Elements with class "title" inside elements with class "post"
article p - All paragraphs inside article elements
.content img - All images inside elements with class "content"

Recent Scrapes

News | UK Society for Co-operative Studies

Scraped at: Unknown
Exclude selector: .listing-results__summary, .cta-page-right-sticky, .page-body__two-col__sidebar, .app__footer, footer, header, nav, aside, .footer, .header, .nav, .navbar, .navigation, .sidebar, .ads, .advertisement, .ad-container, .ad-wrapper, .ad-banner, .cookie-notice, .cookie-banner, .privacy-notice, .gdpr-notice, .popup, .modal, .overlay, .lightbox, .social-share, .social-media, .share-buttons, .social-icons, .comments, .comment-section, .comment-area, .related-posts, .related-articles, .recommended, .breadcrumb, .breadcrumbs, .breadcrumb-nav, .pagination, .page-nav, .pager, .search-box, .search-form, .search-widget, .newsletter, .subscribe, .email-signup, .back-to-top, .scroll-top, .top-button, .menu, .main-menu, .primary-menu, .site-menu, footer, header, nav, aside, .footer, .header, .nav, .navbar, .navigation, .sidebar, .ads, .advertisement, .ad-container, .ad-wrapper, .ad-banner, .cookie-notice, .cookie-banner, .privacy-notice, .gdpr-notice, .popup, .modal, .overlay, .lightbox, .social-share, .social-media, .share-buttons, .social-icons, .comments, .comment-section, .comment-area, .related-posts, .related-articles, .recommended, .breadcrumb, .breadcrumbs, .breadcrumb-nav, .pagination, .page-nav, .pager, .search-box, .search-form, .search-widget, .newsletter, .subscribe, .email-signup, .back-to-top, .scroll-top, .top-button, .menu, .main-menu, .primary-menu, .site-menu, .pagination, .holder, #"primary", #"ribbon", ."scroll", footer, header, aside, .footer, .header, .sidebar, .ads, .advertisement, .ad-container, .ad-banner, .cookie-notice, .cookie-banner, .cookie-popup, .cookie-consent, .privacy-notice, .privacy-banner, .gdpr-notice, .gdpr-banner, .social-share, .social-media, .share-buttons, .social-icons, .comments, .comment-section, .comment-area, .breadcrumb, .breadcrumbs, .pagination, .page-nav, .newsletter, .subscribe, .back-to-top, .scroll-top, [id="cookie-notice"], [id="privacy-notice"], [id="gdpr-notice"], [class="cookie-banner"], [class="privacy-banner"], [class="gdpr-banner"], .alert, .alert-warning, .alert-danger, .alert-success, .alert-info, .alert-wide, .alert-content, .beta-banner, .beta-notice, .beta-alert
🌐 Used browser engine

View Results (Legacy)

News | UK Society for Co-operative Studies

Scraped at: Unknown
Exclude selector: .listing-results__summary, .cta-page-right-sticky, .page-body__two-col__sidebar, .app__footer, footer, header, nav, aside, .footer, .header, .nav, .navbar, .navigation, .sidebar, .ads, .advertisement, .ad-container, .ad-wrapper, .ad-banner, .cookie-notice, .cookie-banner, .privacy-notice, .gdpr-notice, .popup, .modal, .overlay, .lightbox, .social-share, .social-media, .share-buttons, .social-icons, .comments, .comment-section, .comment-area, .related-posts, .related-articles, .recommended, .breadcrumb, .breadcrumbs, .breadcrumb-nav, .pagination, .page-nav, .pager, .search-box, .search-form, .search-widget, .newsletter, .subscribe, .email-signup, .back-to-top, .scroll-top, .top-button, .menu, .main-menu, .primary-menu, .site-menu, footer, header, nav, aside, .footer, .header, .nav, .navbar, .navigation, .sidebar, .ads, .advertisement, .ad-container, .ad-wrapper, .ad-banner, .cookie-notice, .cookie-banner, .privacy-notice, .gdpr-notice, .popup, .modal, .overlay, .lightbox, .social-share, .social-media, .share-buttons, .social-icons, .comments, .comment-section, .comment-area, .related-posts, .related-articles, .recommended, .breadcrumb, .breadcrumbs, .breadcrumb-nav, .pagination, .page-nav, .pager, .search-box, .search-form, .search-widget, .newsletter, .subscribe, .email-signup, .back-to-top, .scroll-top, .top-button, .menu, .main-menu, .primary-menu, .site-menu, .pagination, .holder, #"primary", #"ribbon", footer, header, aside, .footer, .header, .sidebar, .ads, .advertisement, .ad-container, .ad-banner, .cookie-notice, .cookie-banner, .cookie-popup, .cookie-consent, .privacy-notice, .privacy-banner, .gdpr-notice, .gdpr-banner, .social-share, .social-media, .share-buttons, .social-icons, .comments, .comment-section, .comment-area, .breadcrumb, .breadcrumbs, .pagination, .page-nav, .newsletter, .subscribe, .back-to-top, .scroll-top, [id="cookie-notice"], [id="privacy-notice"], [id="gdpr-notice"], [class="cookie-banner"], [class="privacy-banner"], [class="gdpr-banner"], .alert, .alert-warning, .alert-danger, .alert-success, .alert-info, .alert-wide, .alert-content, .beta-banner, .beta-notice, .beta-alert
🌐 Used browser engine

View Results (Legacy)

News | UK Society for Co-operative Studies

Scraped at: Unknown
Exclude selector: .listing-results__summary, .cta-page-right-sticky, .page-body__two-col__sidebar, .app__footer, footer, header, nav, aside, .footer, .header, .nav, .navbar, .navigation, .sidebar, .ads, .advertisement, .ad-container, .ad-wrapper, .ad-banner, .cookie-notice, .cookie-banner, .privacy-notice, .gdpr-notice, .popup, .modal, .overlay, .lightbox, .social-share, .social-media, .share-buttons, .social-icons, .comments, .comment-section, .comment-area, .related-posts, .related-articles, .recommended, .breadcrumb, .breadcrumbs, .breadcrumb-nav, .pagination, .page-nav, .pager, .search-box, .search-form, .search-widget, .newsletter, .subscribe, .email-signup, .back-to-top, .scroll-top, .top-button, .menu, .main-menu, .primary-menu, .site-menu, footer, header, nav, aside, .footer, .header, .nav, .navbar, .navigation, .sidebar, .ads, .advertisement, .ad-container, .ad-wrapper, .ad-banner, .cookie-notice, .cookie-banner, .privacy-notice, .gdpr-notice, .popup, .modal, .overlay, .lightbox, .social-share, .social-media, .share-buttons, .social-icons, .comments, .comment-section, .comment-area, .related-posts, .related-articles, .recommended, .breadcrumb, .breadcrumbs, .breadcrumb-nav, .pagination, .page-nav, .pager, .search-box, .search-form, .search-widget, .newsletter, .subscribe, .email-signup, .back-to-top, .scroll-top, .top-button, .menu, .main-menu, .primary-menu, .site-menu, .pagination, .holder, #"primary", footer, header, aside, .footer, .header, .sidebar, .ads, .advertisement, .ad-container, .ad-banner, .cookie-notice, .cookie-banner, .cookie-popup, .cookie-consent, .privacy-notice, .privacy-banner, .gdpr-notice, .gdpr-banner, .social-share, .social-media, .share-buttons, .social-icons, .comments, .comment-section, .comment-area, .breadcrumb, .breadcrumbs, .pagination, .page-nav, .newsletter, .subscribe, .back-to-top, .scroll-top, [id="cookie-notice"], [id="privacy-notice"], [id="gdpr-notice"], [class="cookie-banner"], [class="privacy-banner"], [class="gdpr-banner"], .alert, .alert-warning, .alert-danger, .alert-success, .alert-info, .alert-wide, .alert-content, .beta-banner, .beta-notice, .beta-alert
🌐 Used browser engine

View Results (Legacy)

News | UK Society for Co-operative Studies

Scraped at: Unknown
Exclude selector: .listing-results__summary, .cta-page-right-sticky, .page-body__two-col__sidebar, .app__footer, footer, header, nav, aside, .footer, .header, .nav, .navbar, .navigation, .sidebar, .ads, .advertisement, .ad-container, .ad-wrapper, .ad-banner, .cookie-notice, .cookie-banner, .privacy-notice, .gdpr-notice, .popup, .modal, .overlay, .lightbox, .social-share, .social-media, .share-buttons, .social-icons, .comments, .comment-section, .comment-area, .related-posts, .related-articles, .recommended, .breadcrumb, .breadcrumbs, .breadcrumb-nav, .pagination, .page-nav, .pager, .search-box, .search-form, .search-widget, .newsletter, .subscribe, .email-signup, .back-to-top, .scroll-top, .top-button, .menu, .main-menu, .primary-menu, .site-menu, footer, header, nav, aside, .footer, .header, .nav, .navbar, .navigation, .sidebar, .ads, .advertisement, .ad-container, .ad-wrapper, .ad-banner, .cookie-notice, .cookie-banner, .privacy-notice, .gdpr-notice, .popup, .modal, .overlay, .lightbox, .social-share, .social-media, .share-buttons, .social-icons, .comments, .comment-section, .comment-area, .related-posts, .related-articles, .recommended, .breadcrumb, .breadcrumbs, .breadcrumb-nav, .pagination, .page-nav, .pager, .search-box, .search-form, .search-widget, .newsletter, .subscribe, .email-signup, .back-to-top, .scroll-top, .top-button, .menu, .main-menu, .primary-menu, .site-menu, .pagination, .holder, footer, header, aside, .footer, .header, .sidebar, .ads, .advertisement, .ad-container, .ad-banner, .cookie-notice, .cookie-banner, .cookie-popup, .cookie-consent, .privacy-notice, .privacy-banner, .gdpr-notice, .gdpr-banner, .social-share, .social-media, .share-buttons, .social-icons, .comments, .comment-section, .comment-area, .breadcrumb, .breadcrumbs, .pagination, .page-nav, .newsletter, .subscribe, .back-to-top, .scroll-top, [id="cookie-notice"], [id="privacy-notice"], [id="gdpr-notice"], [class="cookie-banner"], [class="privacy-banner"], [class="gdpr-banner"], .alert, .alert-warning, .alert-danger, .alert-success, .alert-info, .alert-wide, .alert-content, .beta-banner, .beta-notice, .beta-alert
🌐 Used browser engine

View Results (Legacy)

News

View Results (Legacy)