Data Source Compliance Audit
Review of all 138 source domains indexed by GWSeeker for robots.txt compliance and Terms of Service risk.
138
Compliant
0
Review Needed
0
High Risk / Disable
138
Total Sources
✅ High-Risk Sources Disabled — No Immediate Action Required
All 4 high-risk banking sources have been disabled in the database and will no longer be crawled:
- wellsfargo.com — Disabled (3 sources). WF ToS §3 prohibits automated scraping.
- bankofamerica.com — Disabled (2 sources). BofA ToS prohibits automated access.
- texascapitalbank.com — Disabled. Banking ToS prohibits automated scraping.
- broadway.bank — Disabled. Banking ToS prohibits automated scraping.
Live Robots.txt Compliance Check
Fetches robots.txt for all active HarvesterSource records and checks for blocked paths.
Complete Source Directory
138 sources shown
VA (department.va.gov)
department.va.gov · Federal Agency
VA (va.gov)
va.gov · Federal Agency
USDA Rural Development (rd.usda.gov)
rd.usda.gov · Federal Agency
USDA NRCS (nrcs.usda.gov)
nrcs.usda.gov · Federal Agency
USDA NIFA (nifa.usda.gov)
nifa.usda.gov · Federal Agency
Grants.gov (grants.gov)
grants.gov · Federal Portal
Simpler Grants (simpler.grants.gov)
simpler.grants.gov · Federal Portal
EDA (eda.gov)
eda.gov · Federal Agency
EDA SF Grants (sfgrants.eda.gov)
sfgrants.eda.gov · Federal Portal
FEMA (fema.gov)
fema.gov · Federal Agency
SBA (sba.gov)
sba.gov · Federal Agency
Dept of Education (ed.gov)
ed.gov · Federal Agency
NSF (nsf.gov)
nsf.gov · Federal Agency
IHS (ihs.gov)
ihs.gov · Federal Agency
Dept of Interior (doi.gov)
doi.gov · Federal Agency
CISA (cisa.gov)
cisa.gov · Federal Agency
HUD (hud.gov)
hud.gov · Federal Agency
NEH (neh.gov)
neh.gov · Federal Agency
AmeriCorps (americorps.gov)
americorps.gov · Federal Agency
FHWA (highways.dot.gov)
highways.dot.gov · Federal Agency
USAC (usac.org)
usac.org · Federal Program
USCCB (usccb.org)
usccb.org · Religious Organization
TPWD (tpwd.texas.gov)
tpwd.texas.gov · TX State Agency
TWDB (twdb.texas.gov)
twdb.texas.gov · TX State Agency
TxDOT (txdot.gov)
txdot.gov · TX State Agency
TCEQ (tceq.texas.gov)
tceq.texas.gov · TX State Agency
TX GLO (glo.texas.gov)
glo.texas.gov · TX State Agency
TX Governor's Office (gov.texas.gov)
gov.texas.gov · TX State Agency
TWC (twc.texas.gov)
twc.texas.gov · TX State Agency
TX Comptroller (comptroller.texas.gov)
comptroller.texas.gov · TX State Agency
TX Arts Commission (arts.texas.gov)
arts.texas.gov · TX State Agency
TX Forest Service (tfsweb.tamu.edu)
tfsweb.tamu.edu · TX State / University
TX Agriculture (texasagriculture.gov)
texasagriculture.gov · TX State Agency
TX State Library (tsl.texas.gov)
tsl.texas.gov · TX State Agency
TX Historical Commission (thc.texas.gov)
thc.texas.gov · TX State Agency
Humanities Texas (humanitiestexas.org)
humanitiestexas.org · TX State Nonprofit
TX eGrants (egrants.gov.texas.gov)
egrants.gov.texas.gov · TX State Portal
TX SmartBuy (txsmartbuy.gov)
txsmartbuy.gov · TX State Portal
OneStar Foundation (onestarfoundation.org)
onestarfoundation.org · Nonprofit
Bob Woodruff Foundation
bobwoodrufffoundation.org · Foundation
Gary Sinise Foundation
garysinisefoundation.org · Foundation
Walmart Foundation (walmart.org)
walmart.org · Corporate Foundation
Trull Foundation
trullfoundation.org · Foundation
Tocker Foundation
tocker.org · Foundation
National Trust (savingplaces.org)
savingplaces.org · Nonprofit (National Trust)
Greater Texas Foundation
greatertexasfoundation.org · Foundation
FFCF
ffcf.org · Foundation
LISC
lisc.org · Nonprofit
Anderson Foundation
theandersonfoundation.org · Foundation
JT Hershey Foundation
jthershey.org · Foundation
Bivins Foundation
bivinsfoundation.org · Foundation
Texas Historical Foundation
texashistoricalfoundation.org · Foundation
Entergy Foundation
entergy.com · Corporate
T-Mobile Hometown
t-mobile.com · Corporate
King Foundation
kingfoundation.com · Foundation
Piney Woods Foundation
pineywoodsfoundation.org · Foundation
CGMF
cgmf.org · Foundation
ProLiteracy
proliteracy.org · Nonprofit
Rural Tech Fund
ruraltechfund.org · Foundation
Abell-Hanger Foundation
abell-hanger.org · Foundation
DG Literacy
dgliteracy.org · Foundation
Albertsons Foundation
albertsonscompanies.com · Corporate Foundation
PNC Foundation
pnc.com · Corporate Foundation
MFI Foundation
mfi.org · Foundation
Firehouse Subs Foundation
grants.firehousesubs.com · Corporate Foundation
Gottlieb Foundation
gottliebfoundation.org · Foundation
Local Infrastructure Hub
localinfrastructure.org · Nonprofit
Moody Foundation
moodyf.org · Foundation
Greater Fayette
greaterfayette.com · Community Foundation
AJP Foundation
theajp.org · Foundation
SAAF Foundation (saafdn.org)
saafdn.org · Foundation
SAAF Foundation (saafound.org)
saafound.org · Foundation
Ryan Foundation
ryanfoundation.com · Foundation
American Connection Corps
americanconnectioncorps.org · Nonprofit
Najim Foundation
najimfoundation.org · Foundation
Fund for Sacred Places
fundforsacredplaces.org · Nonprofit
TWU (twu.edu)
twu.edu · University
Wish You Well Foundation
wishyouwellfoundation.org · Foundation
Eske Barrett Foundation
eskebarrettfoundation.org · Foundation
Summerlee Foundation
summerlee.org · Foundation
Preservation Texas
preservationtexas.org · Nonprofit
TLL Temple Foundation
tlltemple.foundation · Foundation
Hunt Family Foundation
huntfamilyfoundation.com · Foundation
US Chamber Foundation
uschamberfoundation.org · Foundation
Vilcek Foundation
vilcek.org · Foundation
Owen Family Foundation
owenfamilyfoundation.org · Foundation
Scharbauer Foundation
scharbauerfoundation.org · Foundation
Rapoport Foundation
rapoportfdn.org · Foundation
Priddy Foundation
priddyfdn.org · Foundation
Fasken Foundation
faskenfoundation.org · Foundation
PBAF
pbaf.org · Foundation
VBLF
vblf.org · Foundation
Behmann Brothers Foundation
behmannbrothersfoundation.org · Foundation
Stanzel Family Foundation
stanzelfamilyfoundation.org · Foundation
CBC Foundation
cbcfoundation.org · Foundation
Texoma Health Foundation
texomahealth.org · Foundation
Episcopal Health Foundation
episcopalhealth.org · Foundation
St. David's Foundation
stdavidsfoundation.org · Foundation
Community Foundation Abilene
cfabilene.org · Community Foundation
Amarillo Area Foundation
amarilloareafoundation.org · Community Foundation
Austin Community Foundation
austincf.org · Community Foundation
CF Brazos Valley
cfbv.org · Community Foundation
CF West Texas
cfwtx.org · Community Foundation
Central TX CF
ctxcf.org · Community Foundation
United Way Comal
uwct.org · United Way
LCRA
lcra.org · Public Utility
Levitt Foundation
levitt.org · Foundation
Eastman Foundation
eastman.com · Corporate
Community Foundation Network
communityfoundation.net · Community Foundation
AARP Community Challenge
aarp.org · Nonprofit
VIA Art Fund
viaartfund.org · Foundation
Sundt Foundation
sundt.com · Corporate
Sparklight Giving
sparklight.com · Corporate
CoServ Foundation
coserv.com · Corporate
Texas Mutual
texasmutual.com · Corporate
WFACF
wfacf.org · Community Foundation
BCBS Texas
bcbstx.com · Corporate
CenterPoint Foundation
centerpointenergy.com · Corporate
EP Community Foundation
epcf.org · Community Foundation
Hogg Foundation (UT)
hogg.utexas.edu · University Foundation
Sid Richardson Foundation
sidrichardson.org · Foundation
Texas Methodist Foundation
texasmethodistfoundation.org · Foundation
Isla Carroll Turner Trust
islacarrollturnerfriendshiptrust.org · Foundation
Aspen Institute
aspeninstitute.org · Nonprofit
Sewell Fund
sewellfund.org · Foundation
Sterling Turner Foundation
sterlingturnerfoundation.org · Foundation
JW Couch Foundation
jwcouchfoundation.org · Foundation
Texas Rural Foundation
texasrural.org · Nonprofit
Grantmakers.io (aggregator)
grantmakers.io · Aggregator
Texas Book Festival
texasbookfestival.org · Nonprofit
TXLA
txla.org · Professional Assoc.
ALA
ala.org · Professional Assoc.
BSA Community Foundation
bsacf.org · Community Foundation
Brookshire's Community
brookshires.com · Corporate
The Classroom Fund
theclassroomfund.org · Nonprofit
Community Care Today
communitycare.today · Nonprofit
Buc-ee's
buc-ees.com · Corporate
AEP Community Giving
aep.com · Corporate
Compliance Framework Summary
✅ What We Do Right
- robots.txt is checked before every crawl (both
crawlBatchandweeklyCrawl) - 1-second polite delay between sub-page requests
- Custom User-Agent header:
GWSeeker-Crawler/1.0(identifiable, not spoofed) - Only public-facing grant information is extracted — no login-gated content
- Federal and state government sites are explicitly designed for public access
- LLM extraction reads pages publicly accessible to any browser user
- All high-risk banking sources (Wells Fargo, BofA, TX Capital Bank, Broadway Bank) disabled in database
🔧 Recommended Actions
- Add a ToS acknowledgment field to HarvesterSource entity for audit trail
- Run Live Robots.txt Audit above monthly to catch new blocks
- Periodically re-verify high-risk banking sources remain disabled in the database