I read some years ago that elderberry worked better than a flu shot to prevent flu in seniors in one study. The seniors' immune system couldn't muster up enough "oomph" to have the flu shot be effective, but elderberry works by preventing the virus from being able to latch onto healthy cells in the first place (so it can't replicate).
So by all means have at the elderberry! But your immune system does seem in need of strengthening in general. Nutrition, exercise, clean air, de-stress...you know the kind of thing! You could also try ginseng tea as a tonic for your immune system.