Fail to extend a Stream with new methods #442

Activity

1. martindurant commented on Dec 5, 2021

   

   martindurant

   on Dec 5, 2021

   Member

   More actions

   It would work with @Stream.register_api - will that suffice?
2. florentbr commented on Dec 6, 2021

   

   florentbr

   on Dec 6, 2021

   Author

   More actions

   @martindurant, the issue with @Stream.register_api is that it will mutate the original Stream class which will impact all the instances everywere else. So it doesn't fullfill the requirement for extending a stream. For instance, if I were to override the sink method, it would mess/break the other streams. What I'm expecting is basic polymorphism.
3. florentbr commented on Dec 6, 2021

   

   florentbr

   on Dec 6, 2021

   Author

   More actions

   It's called subclassing in the pandas documentation :
   https://pandas.pydata.org/docs/development/extending.html#subclassing-pandas-data-structures
4. martindurant commented on Dec 6, 2021

   

   martindurant

   on Dec 6, 2021

   Last edited by martindurant

   Member

   More actions

   I think I see what you are trying to do, but you face the following problem: map is a method on Stream (and defined by the map class), so that when you call .map, there is no easy way to know that this was called from within a subclass - map is not a child of your custom class and cannot be made so dynamically.

   That leaves you with the two obvious paths, as you point out: come up with unique names for all node types, or allow name clashes and understand that you may clobber for all streams. For your example, I'm not totally sure how you expect the method draw to be called, since it doesn't actually take any events. All existing nodes are implemented at the update (incoming events) and emit (outgoing events) points.

   However, another possibility, is maybe what you are actually after. You can use Stream.connect to tie together nodes without using the register_api route. For example, the following are equivalent, where the second and third versions avoid using the automatically created method.

       s = Stream()
       m = s.map(func)

   or

       s = Stream()
       m = streamz.core.map(s, func)

   or

       s = Stream()
       m = streamz.core.map(None, func)
       s.connect(m)
5. florentbr commented on Dec 7, 2021

   

   florentbr

   on Dec 7, 2021

   Author

   More actions

   map is a method on Stream (and defined by the map class), so that when you call .map, there is no easy way to know that this was called from within a subclass

   map is a wrapped function. The class/subclass is available at the time it's registered and called :

   streamz/streamz/core.py

   Line 161 in 7453431

   return func(*args, **kwargs)

   map is not a child of your custom class and cannot be made so dynamically.

   The following Stream.register_api() would adds the expected inheritance at registration :

       @classmethod
       def register_api(cls, func):
           @functools.wraps(func)
           def wrapped(self, *args, **kwargs):
               clazz = type(func.__name__, (func, type(self)), {})
               node = clazz(self, *args, **kwargs)
               return node
           setattr(Stream, func.__name__, wrapped)

   For your example, I'm not totally sure how you expect the method draw to be called, since it doesn't actually take any events.

   This is just an example to demonstrate the issue with inheritance. Don't read too much into it.

   You can use Stream.connect to tie together nodes without using the register_api route.

   This library threats it's methods as plugings but doesn't keep track of them.

   So I'd have to rewrite all the methods that are registered by Stream.register_api to connect the nodes. It kind of defeat the purpose of inheritance.
6. martindurant commented on Dec 7, 2021

   

   martindurant

   on Dec 7, 2021

   Member

   More actions

   I haven't had a chance yet to understand what difference your alternate decorator has, but please do propose a PR to demonstrate it! I daresay there are others who have thought along these lines and not known how to proceed.
7. florentbr

   added 4 commits that reference this issue on Dec 7, 2021

   Fix Stream subclassing python-streamz#442

   45edc86

   Fix Stream subclassing python-streamz#442

   6aeb4ee

   Add Stream node constructor for sub-classing python-streamz#442

   0a8083c

   Add Stream node constructor for sub-classing python-streamz#442

   f9d993c