
Subsets
I find subsets, as presented in "foundations of analysis" type books, confusing. They start off with a definition of subset as "if x in A then x in B."
Later the discussion moves to subsets of a Euclidean space and open subsets. Is a surface S in R3 a subset of R3? What is the neighborhood of a subset of S? What is the interior of a subset of S? If the neighborhood is defined in R3, there is no such thing as an open surface as a subset in Euclidean space.
What implications does this have for a mapping from R3 to a subset of R3 if the subset is a surface?
In researching the question, I came up with the following:

A topological space is a set X together with a collection T of subsets of X satisfying the following axioms:
 The empty set and X are in T.
 The union of any collection of sets in T is also in T.
 The intersection of any pair of sets in T is also in T.
The set T is a topology on X. The sets in T are the open sets, and their complements in X are the closed sets. The elements of X are called points.
 More generally, the Euclidean spaces Rn are topological spaces, and the open sets are generated by open balls.
 Any metric space turns into a topological space if one defines the open sets to be generated by the set of all open balls.
A function between topological spaces is said to be continuous if the inverse image of every open set is open.
http://www.wordiq.com/definition/Topological_space

which is a marvel of clarity. The only question remaining is, can you talk about continuity for a function which maps a "3dimensional" subset of R3, say a solid sphere, into a line or surface in R3?

The original definition of subset that you have is correct. A subset of R^3 is just any collection of points from R^3. For example, lines, planes, and arbitrary surfaces are subsets of R^3 because they are just collections of points.
I'm not sure what the definition of a neighborhood of a subset is, but a neighborhood of a point is just an open ball containing that point. Perhaps a neighborhood of a subset is a union of open balls over all the points in the subset.
The interior of a subset is the collection of points from the subset that have a neighborhood fully contained in the subset.

Open Ball and Subset
DrSteve is right. A subset is clear. But what is an open subset, which seems to be a core concept of Topology.
Open Ball: The set of all points in a metric space E whose distance from x is less than r.
Open Subset S in E: For each x in S, S contains an Open Ball centered at x. By this definition a plane cannot be a subset of R3.
Open Ball in an arbitrary subset S of E: The set of points in S of an open Ball in E whose center is in S. In this case the subset is considered a subspace of E, ie, distance between points in S defined as points in E. 1)
Example: A warped surface S in R3:
1) Distance between points is the distance between the points in R3. An open ball in S is the intersection of an open ball in R3 centered at x in S, with S.
2) Same warped surface without any reference to R3. Coordinate lines scratched on surface with numbers x1,x2 attached to lines. Any point in the surface is then given by (x1,x2). To make this a metric space you have to define the distance between two points, not easy. A "ball" is also difficult to define. However an open neighborhood, and therefore open subsets, can be defined without a metric by using a "box:" a<x1<b, c<x2<d.
Reference: First Chapters of:
1) Rosenlicht, Introduction to Analysis.
2) Shilov, Elementary Real and Complex Analysis.
3) Taylor, General Theory of Functions and Integration.