Sorry about the Houttuynia cordata - chameleon plant. You will find the goutweed was easier to get rid of! I'd say dig it out and trash it.
Consider looking at some books on woodland gardens. Many woodland plants are ephemeral while some have lovely leaves that will give you color throughout the summer.
I'm confused with your hardiness zone notation. How can you be in zones 4 and 6 at the same time? Maybe a hardiness zone map will help.
Do check the hardiness zones of these plants.
Heuchera - coral bells, a clump forming native, comes in an array of colorful leaves for color all season long. One of my favorite combos of these would be a group of Heuchera 'Lime Rickey' with Heuchera 'Plum Pudding' in front of them.
Tiarella - foam flower, another clump forming native, comes with pretty colored leaves. This first one is called Tiarella 'Neon Lights'. The third pic is a grouping of the plain green leaves, but would look lovely with something in front or behind with colored leaves.
Corydalis - comes in different colored flowers, seeds around the garden, is easy to pull out and will appear in wonderful spots all by itself.
Tricyrtis - toad lily has dainty flowers and blooms in fall. Place near the front of a border or the edge of a path so they can be seen. Best to plant at least 5 together for better impact.
Asarum - woodland ginger has lovely heart shaped leaves either green or variegated. Makes a good groundcover in shady places. Native and non-natives.
There's all kinds of ferns. Some may stay evergreen for you.
A mix of Japanese painted ferns, sedge (the grassy looking stuff), maidenhair fern in the back and a Heuchera 'Plum Pudding' on the right.
More mix with a Heuchera in bloom, a green hosta and astilbe on the extreme left with the small leaves and a plume from an astilbe on the extreme right.
My astilbe wants more sun and blooms better with at least 4 hours of sun.
Bergenia is evergreen for me but don't like the soil too rich. I love the glossy large leaves.
Ligularia - many different varieties and different cultivars. They wilt midday but perk back up in the evening. They like it shady and a bit moist. Ligularia 'the rocket' is quite a statement when it blooms and they get large. Ligularia dentata has flowers that look like black eyed susans.
Bleeding heart in pink or white. My old fashioned ones bloom for a long time in a moist spot and form a lovely clump. In a dry spot they go dormant mid summer. They look nice with larger leaved hostas. There is a new smaller one called fern-leaf bleeding heart. The entire plant is smaller and blooms on and off all summer.
Helleborus - Lenten rose in many colors and should be evergreen.
Pulmonaria - Lungwort is another shade lover with incredible variegation. Many cultivars, some better at resisting powdery mildew then others. The flowers change from pink to blue as they age in the spring. I have 'Mrs Moon' and it's more mildew resistant and an old cultivar.
Solomon's seal will form a lovely clump. There's an all green native - False Solomon's seal and a Japanese variegated one that's a true Solomon's seal. Mine are mixed with hostas and ferns. Watch the berries on these if you have small kids.
Phlox divaricata - Woodland phlox is a native groundcover with different colored flowers. The most common is blue. Spreads without being invasive and likes rich soil. Nice with ferns growing through it. This is not the tall fragrant phlox.
Mitchella repens - Partridgeberry A slow grower that forms a small mat of evergreen foliage. Height 2-4 in. Pink-white flowers in June with red berries that last into winter. Prefers moist soil.
Iris cristata - Dwarf Crested Iris creates a nice ground cover when grown in a mass. Easy to grow. Height: 6-8 in. Blue-violet flowers in May. A nice contrast to broad leaved plants. Grows best in fertile, well-drained soil with a few hours of sun.
Epimediums aren't evergreen in your zone. Several colors and leaf shapes. Just in case you can't pick just one...
or you could look at these 5 pages!
Lamium takes dry shade, but be careful with these as they can spread out of hand. If you choose one and it says 'vigorous' you'll be pulling it out of your lawn as well as pulling your hair out.
Ideas for mixing plants under trees.
Mostly hostas and ferns
Mostly hostas with epimedium in the front.
When my pond is finished I'm going to plant Hakonechloa so it cascades over the rocks. This looks great at the front of a bed. Comes in different shades of variegation. Looks great with solid green hostas, purple leaved plants (Ligularia) or large ferns behind it.
Some nice combos.
You can click your way through these plants and research them at www.google.com
Best to use the botanical names. Also click on 'Images' at google to get more pics.
If you decide to mailorder hold on to this site to check references and search for highly rated nurseries near you. You can search by country and then by plant material.
Ok, I've now eaten too many cookies and drank too much milk and don't know if I can go to sleep. Don't stay up all night reading this!