You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Comparison of three submissions against the Support Engineer Task requirements.
Each submission was reviewed line-by-line, tracing every execution path for correctness, logic errors, security issues, and adherence to the spec.
Nominally (embedded list), but $set on entire versions array means concurrent update can overwrite
Rating
5/10
7/10
4/10
4. Nice to Haves
Criteria
lady-thee
TijanAyo
ruggedwizard
Unit tests
None
~63 unit + 8 integration tests across 8 files
None
Test quality
N/A
Several issues: try/catch without expect.assertions() (false positive risk), shared mutable state across tests (order-dependent), missing edge cases (NaN, empty schema, concurrent operations, mathjs functions in expressions)
N/A
Error formatting
Partial (flat dict for submissions, string list for schema)
Best: field-level error arrays with human-readable messages using field labels, proper HTTP status codes
Partial, but errors overwrite each other (only last error per field shown)
Documentation
Good README (endpoints, examples, field types)
Minimal README (no request/response examples, no schema format docs)
Good README (cURL examples), but references .env.sample that doesn't exist
Docker/infra
No
Yes (Docker Compose with MongoDB + mongo-express UI)
No
Installability
Broken (requirements.txt is UTF-16 encoded, pip install fails)
Works
Works
Unused dependencies
email-validator installed but hand-rolled regex used instead, sentry-sdk installed but never configured
@types/mathjs deprecated and conflicts with mathjs v15
jsonschema in requirements but never imported
Rating
2/10
7/10
2/10
5. Security
Vulnerability
lady-thee
TijanAyo
ruggedwizard
Remote Code Execution
eval() with {"__builtins__": {}} is trivially bypassable via ().__class__.__bases__[0].__subclasses__()
mathjs.evaluate() is powerful but more limited than Python's eval. Still, mathjs docs explicitly warn it's not sandboxed.
Same as lady-thee, plus whitelisted builtins (abs, min, max, round) expand attack surface
Internal info leakage
Unhandled exceptions expose stack traces
All errors return 400 with raw err.message, leaks Mongoose CastErrors, MongoDB ObjectIds, internal schema details
Custom exception hierarchy (better)
Request validation
Pydantic validates request bodies (good)
Zero runtime request validation. Malformed body crashes with raw TypeError.
Pydantic validates request bodies (good)
Extra field injection
Unvalidated extra fields stored in DB
Unvalidated extra fields stored in DB
Unvalidated extra fields stored in DB
No auth/rate limiting/CORS
Yes
Yes
Yes
Rating
3/10
4/10
3/10
6. Code Quality and API Design
Criteria
lady-thee
TijanAyo
ruggedwizard
Architecture
Layered (routes, services, validators, models)
Layered (models, services, utils, routes)
God class (FormEngine handles CRUD, validation, submission, normalization, computed evaluation)
Cannot retrieve form schema
No endpoint returns the form schema/field definitions. Consumers cannot see what fields a form has.
Schema returned in create/update responses
Schema returned in responses
HTTP status codes
Appropriate (400, 404, 422)
All errors return 400 regardless of type (not-found, validation, server error)
Appropriate (400, 404, 422)
Runtime crashes on basic operations
GET /api/forms crashes every time (.model_dump() on list). Number validation crashes on every number field with min/max.
Malformed request body crashes (no runtime validation)
gt/lt condition with type mismatch crashes
DB connection handling
Missing env vars silently connect to localhost with DB named "None"
Proper connection wrapper
Motor client created at module import (before event loop)
DB indexes
None
None
None
N+1 queries
Yes (get_forms queries each version separately)
Yes (form fetched twice in submit)
No
TypeScript type safety
N/A
strict: true but defeated by Mixed schemas, any casts in catches, as string/as number unsafe casts, skipLibCheck: true
N/A
Rating
3/10
5/10
3/10
7. Summary Scores
Category
Weight
lady-thee
TijanAyo
ruggedwizard
Core Models
10%
4/10
7/10
6/10
create_form
12%
6/10
6/10
6/10
update_form
10%
4/10
7/10
6/10
validate_submission
15%
3/10
6/10
4/10
submit
10%
4/10
6/10
5/10
all_submissions
10%
5/10
5/10
3/10
Forward/Backward Compatibility
8%
5/10
7/10
4/10
Nice to Haves
10%
2/10
7/10
2/10
Security
7%
3/10
4/10
3/10
Code Quality & API Design
8%
3/10
5/10
3/10
Weighted Total
100%
3.87/10
6.02/10
4.18/10
8. Final Ranking
Rank
Candidate
Score
Verdict
1
TijanAyo
6.02/10
Best of the three, but far from production-ready. Key strengths: only candidate with tests (~63 unit + 8 integration), uses mathjs instead of eval(), cleanest model design (separates computed data), richest conditional logic, Docker Compose, TypeScript strict mode. Key weaknesses: TS type safety is a facade over Mixed schemas, zero runtime request validation (malformed body crashes), all errors return 400, expression regex rejects all mathjs functions, computed fields can't reference other computed fields, race condition between validation and submission, stale computed values in normalization.
2
ruggedwizard
4.18/10
Addresses all five requirements, but the implementation has deep issues. The FormEngine god class does everything. all_submissions has a bare except: pass and retroactively recalculates computed values (corrupts historical data). Validation errors overwrite each other (only last error per field). min_length=0 truthy bug. Condition operators are unvalidated strings. gt/lt comparisons crash on type mismatch. eval() for computed fields (RCE). No tests.
3
lady-thee
3.87/10
All requirements addressed structurally, but multiple critical runtime failures make core features non-functional. GET /api/forms crashes on every call (.model_dump() on list). Number min/max validation crashes (wrong attribute names). All timestamps are identical (class-level datetime.now()). requirements.txt is corrupted (project can't be installed). No endpoint to retrieve form schemas. Version off-by-one in update response. eval() for computed fields (RCE). No tests.
9. Issue Catalog by Candidate
lady-thee (helios-form): 58 issues
Critical runtime crashes (7)
datetime.now() evaluated at class definition time; all documents share server-start timestamp
_validate_number accesses field.min/field.max which don't exist (model defines min_value/max_value); crashes with AttributeError
GET /api/forms calls .model_dump() on a List[FormResponse]; crashes with AttributeError every time
eval() with {"__builtins__": {}} is trivially bypassable for remote code execution
get_forms_service compares PydanticObjectId to str in version lookup; never matches, version always reports 0
update_form_service response reports version N+2 instead of N+1 (off-by-one)
PydanticObjectId(None) crashes when latest_version_id is None (multiple locations)
NaN passes all number validation (typeof NaN === "number" is true)
Table column types completely ignored (column type field never validated)
Table rows accept extra columns beyond schema definition
Empty string skips all validation for non-required fields (problematic for non-required numbers)
Extra/unknown fields in submission silently accepted and stored
SubmissionService logic bugs (7)
Form fetched twice in submit (once in validateSubmission, once in submit)
Race condition: concurrent updateForm between validation and save causes version mismatch
evaluateAll failure crashes entire submission (exposes schema internals in error)
allSubmissions serves stale computed values; new computed fields become null
No pagination on allSubmissions (loads ALL submissions into memory)
O(nmm) field lookups in normalization (.find() inside nested loops)
Non-null assertion ! on potentially undefined value
Expression and visibility evaluator issues (6)
mathjs.evaluate can return non-number types (matrices, complex numbers, booleans)
mathjs can mutate the scope object (e.g., expression "x = 42" modifies submission data in memory)
gt/lt silently return false for non-number values (no error)
eq uses strict equality (fails for string/number comparisons, fails for object/array comparisons)
neq with undefined field makes field visible (asymmetric with eq behavior)
Default case returns true (unknown operators silently make fields visible)
Schema diff issues (2)
Duplicate field names cause silent data loss in Map
JSON.stringify comparison is key-order sensitive (false "modified" detections)
Route handler issues (2)
PUT/POST/GET return 400 for "not found" instead of 404
All internal server errors return 400 instead of 500 (leaks internals)
Test quality issues (8)
try/catch without expect.assertions() allows silent false positives
No test for invalid ObjectId format
No test for concurrent operations / race conditions
No tests for expression edge cases (mathjs functions, assignment operators, undefined vars)
No test for NaN number values
No test for empty schema
Integration test doesn't verify stale computed value behavior
SubmissionService tests are order-dependent (shared mutable state)
Model and infrastructure issues (7)
No index on Form.name (no uniqueness constraint)
No compound index on FormVersion(formId, version) (race condition can create duplicate versions)
No index on Submission.formId (full collection scan)
Inconsistent timestamp handling (timestamps: true vs manual submittedAt)
No graceful shutdown handling (SIGTERM/SIGINT)
No global error handling middleware
No 404 handler for unmatched routes
ruggedwizard (support-assessment): 29 issues
Critical (1)
eval() with restricted builtins is trivially bypassable for remote code execution
Runtime crashes (2)
greater_than/less_than condition operators raise TypeError on type mismatch
README references .env.sample that doesn't exist
Silent wrong results (5)
errors[f.name] = ... overwrites previous errors (only last error per field survives)
min_length=0 treated as "no constraint" due to truthy check
Bare except: pass swallows ALL errors including SystemExit/KeyboardInterrupt
all_submissions retroactively recalculates computed fields with new expressions on old data
Unvalidated operator string; typo silently returns False, permanently hiding field
Data integrity (4)
Read-modify-write race in update_form can lose schema versions
Hidden fields skip validation but data stored unvalidated
Dead isdigit() check in expression validation
No transactions on multi-step DB operations
Design/code quality (12)
FormEngine god class
Email validation is "@" in val
Table validation: isinstance(val, list) only
_doc_to_form mutates input dict
form_id stored as string instead of ObjectId
Optional[str] on id fields misleads consumers
datetime.utcnow() deprecated
Triple-quoted strings as section dividers (not docstrings)
Motor client created before event loop
No DB indexes
Hardcoded limit=1000, silently truncates
No-op updates create new versions
Missing (5)
Zero tests
No CORS
No .env.sample
No pagination
No rename tracking in changelog
10. Conclusion
All three submissions have fundamental issues that would prevent production deployment. The most significant finding from the deep review is that issue counts are much higher than surface-level review suggests, and the nature of the bugs reveals how thoroughly (or not) each candidate tested their own code.
lady-thee has bugs that would be caught by running the code even once (GET /api/forms crashes, number validation crashes, corrupted requirements.txt prevents installation). This suggests the code was never executed end-to-end.
TijanAyo has the most issues by count (61), but the majority are edge cases, missing validations, and TypeScript-specific concerns rather than core logic failures. The existence of ~71 tests shows the code was actually run and tested, even though the test suite has its own quality issues. The TypeScript type safety is largely theatrical (Mixed schemas defeat it at the persistence boundary), and the expression regex bug means no mathjs function can be used in computed expressions, which undermines the choice of mathjs.
ruggedwizard's code reads like a first draft: the god class, bare except: pass, error overwriting, truthy bugs, and complete absence of tests all point to code written in one pass without review. The retroactive computed field recalculation in all_submissions is a design-level mistake that actively corrupts data, which is worse than simply losing data.
TijanAyo remains the strongest candidate. Despite the high issue count, the submission demonstrates meaningfully better engineering practices than the other two, and its issues are categorically less severe (edge cases vs. core functionality failures).
